Skip to content

feat: API fetcher service + api_json parser for HTTP polling data ingestion - #52

Merged
cchwala merged 8 commits into
mainfrom
feat/api-fetcher
Jul 13, 2026
Merged

feat: API fetcher service + api_json parser for HTTP polling data ingestion#52
cchwala merged 8 commits into
mainfrom
feat/api-fetcher

Conversation

@cchwala

@cchwala cchwala commented May 20, 2026

Copy link
Copy Markdown
Member

What

Adds automated data ingestion from HTTP/JSON APIs via a new api_fetcher service that polls external endpoints and writes JSON files to the parser's incoming directory. Includes the api_json parser that handles these JSON files.

Architecture

External API → api_fetcher (polls HTTP) → JSON files → parser (api_json) → database

The api_fetcher runs as a separate container, decoupled from the parser. It:

  • Polls configured HTTP endpoints on a schedule
  • Handles authentication (API keys, bearer tokens)
  • Writes normalized JSON to shared incoming volume
  • Tracks polling state to avoid duplicates

The parser watches the same incoming directory and processes JSON files identically to CSV files.

Components

fetchers/shared/ — Shared utilities

  • config.py — Load fetcher config from YAML
  • polling.py — Generic polling loop with backoff
  • state.py — Persist last-polled timestamp to avoid re-fetching
  • incoming_writer.py — Write fetched data to shared volume

fetchers/api_fetcher/ — Main fetcher service

  • main.py — Entrypoint, wires together components
  • fetcher.py — HTTP client with retry logic
  • auth.py — Authentication handlers (API key, bearer token, basic auth)
  • config.yml — Example configuration

parser/parsers/api_json/ — JSON parser

  • parse_raw.py — Parse JSON with field-map-based column routing
  • example_field_map.yml — Longest-prefix matching on filename to route source fields to canonical names (time, cml_id, sublink_id, rsl, tsl)

Integration

  • parser/service_logic.py — Add load_api_json_bundle() helper
  • parser/main.py — Select api_json bundle when PARSER_TYPE=api_json
  • scripts/generate_config.py — Accept api_json as valid parser type

Configuration

Add to deploy repo's compose override:

services:
  api_fetcher_user1:
    build: ./GMDI_prototype/fetchers/api_fetcher
    environment:
      - FETCHER_CONFIG_PATH=/app/config.yml
      - INCOMING_DIR=/app/incoming
    volumes:
      - ./api_fetcher_config.yml:/app/config.yml:ro
      - parser_user1_incoming:/app/incoming

Parser service uses PARSER_TYPE=api_json.

Testing

  • 6 tests in test_api_json_parser.py covering field-map routing, UTC normalization, empty files, error paths
  • Mock server in fetchers/api_fetcher/mock_server/ for integration testing
  • Full pipeline tested end-to-end with api_fetcher → parser → database

Backward compatibility

No breaking changes. Existing CSV parsers continue to work unchanged. The api_json parser is opt-in via PARSER_TYPE=api_json.

cchwala added 5 commits May 20, 2026 23:05
- fetchers/shared: config loader, atomic incoming writer, state
  persistence (cursor + seen-files), poll loop with exponential backoff
- fetchers/api_fetcher: JWT auth with auto-refresh, windowed paginated
  fetcher (continuous + backfill modes, param_variants for RSL/TSL),
  entry point, example config.yml, Dockerfile (build ctx ./fetchers)
- fetchers/api_fetcher/mock_server: minimal Flask mock with JWT auth
  and synthetic hourly CML data (POST /login/, POST /refresh/, GET /cml/)
Parses JSON files produced by api_fetcher.  A field map (loaded from
FIELD_MAP_PATH env var) does longest-prefix matching on the filename
stem to route source fields to [time, cml_id, sublink_id, rsl, tsl].
Includes example_field_map.yml for the mock server (RSL + TSL split).
parser/service_logic.py: import parse_api_json_raw; add elif .json
  branch routing JSON files to write_rawdata (before else: quarantine)
parser/main.py: add .json to FileWatcher extensions; add *.json glob
  in process_existing_files for startup replay
docker-compose.dev.yml: adds mock_server (port 5001) and api_fetcher
  services for local development; mounts shared data volume and
  example field_map.yml
fetcher.py: in continuous mode, end was computed as start+chunk_hours;
  with overlap_seconds >= chunk_hours*3600 (as in the example config) the
  window end equalled the cursor, so the cursor never advanced (infinite
  loop). Fix: end = cursor_dt + chunk_hours; overlap only shifts start back.

parser/main.py: process_existing_files returned early when there were no
  CSV files, skipping the JSON glob entirely. Fix: guard only the CSV
  section, always run the JSON section.

mock_server/app.py + docker-compose.dev.yml: healthcheck hit GET /login/
  which is POST-only; Flask returned 405, urlopen raised HTTPError, check
  always failed → api_fetcher never started. Fix: add GET /health endpoint,
  point healthcheck there.
@codecov

codecov Bot commented May 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.52055% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.21%. Comparing base (5b2c4da) to head (5209788).

Files with missing lines Patch % Lines
parser/main.py 45.45% 6 Missing ⚠️
parser/service_logic.py 83.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #52      +/-   ##
==========================================
+ Coverage   85.86%   86.21%   +0.34%     
==========================================
  Files          35       37       +2     
  Lines        3269     3409     +140     
==========================================
+ Hits         2807     2939     +132     
- Misses        462      470       +8     
Flag Coverage Δ
mno_simulator 86.12% <ø> (ø)
parser 92.20% <94.52%> (+0.16%) ⬆️
scripts 74.13% <ø> (ø)
webserver 73.78% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cchwala added 3 commits May 20, 2026 23:19
service_logic.py: the JSON elif was placed AFTER 'raw'/'data' in name,
  but all api_fetcher filenames end in _data.json and matched the CSV
  branch first, making the JSON branch unreachable. Fix: check .json
  suffix first, before the substring checks.

test_service_logic.py: add test_process_cml_file_json_archived covering
  the repaired JSON branch (patches parse_api_json_raw, asserts write
  and archive calls).

test_api_json_parser.py: new test module for parse_api_json_raw covering
  prefix matching, RSL/TSL column routing, UTC datetime normalisation,
  empty-file handling, and error paths (missing field, no prefix match).
…nfrastructure with API fetcher

Resolution adds api_json support to the new ParserBundle architecture:

- Add load_api_json_bundle() helper in service_logic.py
- Wire api_json parser selection in main.py based on PARSER_TYPE env var
- Update generate_config.py validation to accept 'api_json' as valid parser type
- Import _make_default_bundle in main.py for test compatibility

This preserves all api_fetcher functionality (JSON polling, api_json parser)
while adopting the pluggable parser infrastructure from csv-generic-parser PR.
@cchwala cchwala changed the title Feat/api fetcher feat: API fetcher service + api_json parser for HTTP polling data ingestion Jul 13, 2026
@cchwala
cchwala merged commit be972d2 into main Jul 13, 2026
8 checks passed
@cchwala
cchwala deleted the feat/api-fetcher branch July 13, 2026 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant